Multimedia watermarking system and method

ABSTRACT

A system and method for verifying and authenticating multimedia objects. Provided is a system for splitting a watermark into at least a first and second part, inserting the first part of the watermark into a first component of the multimedia object, and inserting the second part into a second component of the multimedia object, and outputting a watermarked multimedia object. The watermark may be provided by obtaining a signature of the original multimedia object. Also included are systems and methods for checking the validity and authenticity of a received watermarked multimedia object, wherein the multimedia object includes individual watermarked media components.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to verifying and authenticating multimediaobjects, and more particularly to a system and method to combinewatermarks in a plurality of multimedia components.

2. Related Art

Watermarking is the process of hiding or inserting data in multimediaobjects, such as audio, video and still image content. An insertedwatermark can be used for various purposes, including: (1) verification,i.e., to identify the rightful owner of the content and protect thecopyrights therein; and (2) authentication, i.e., to ensure that thecontent has not been subjected to alteration.

In the case of watermarking for verification, the inserted watermark canbe used to identify the owner of the content. Specifically, by insertinga watermark into the content, an owner can later prove ownership byextracting the watermark and showing that it matches the one originallyinserted. In the case of watermarking for authentication, the insertedwatermark can be used to verify the authenticity of the content byidentifying content that has been tampered with. One way to achieve thisis to provide a watermark that is dependent on the content. In thiscase, the watermark is typically a signature (or a function of thesignature) of the content. This signature is chosen so that it reflectsthe salient characteristics of the content. Accordingly, if the contenthas been altered, the original watermark that reflects the originalsignature of the content will not match the new watermark, whichreflects the signature of the tampered content.

Many different techniques for inserting watermarks in multimediacomponents are known. A typical watermark comprises data that canidentify the owner (e.g., a visual logo) for verification; or data thatcaptures the salient visual or auditory characteristics of themultimedia component for content authentication. In many cases, thewatermark is inserted in the content without causing any perceivablechange to the actual audio or video content. Thus, an end-user isgenerally not aware of the watermark.

A common problem, however, relates to ensuring that an insertedwatermark has not been tampered with. For instance, consider the case ofa watermarked video image. In a video image, successive frames tend tobe almost identical due to the lack of motion between them. Such a lackof change between frames provides an opportunity for a hacker toidentify or weaken the inserted watermark. For example, if a given frameis watermarked and the adjacent frame(s) are not watermarked, and arevery similar to the watermarked frame, then the hacker can simplysubtract the watermarked frame from the unwatermarked frame(s) to obtainthe watermark. Identifying the watermark will enable the hacker toremove the watermark and even insert a different watermark to change theownership rights. Even if the successive frames are watermarked (withdifferent watermarks), the hacker can average the successive frames toweaken the watermark.

Accordingly, there exists a need to provide secure watermarking inmultimedia objects, particularly those that include video. Without sucha secure system, multimedia content will be subject to hackers who caneasily defeat inserted watermarks.

SUMMARY OF THE INVENTION

This invention overcomes the above-mentioned problems, as well asothers, by providing a technique to combine the watermarks in individualmultimedia components, such as audio and video. In a first aspect, theinvention includes a method of protecting a multimedia object having afirst media component and a second media component, comprising the stepsof: (1) providing a watermark; (2) splitting the watermark into a firstpart and a second part; (3) inserting the first part of the watermarkinto the first media component; (4) inserting the second part of thewatermark into the second media component; and (5) outputting awatermarked multimedia object.

The watermarked multimedia object can be verified by: (1) receiving thewatermarked multimedia object; (2) extracting from the first mediacomponent of the watermarked multimedia object a first extractedwatermark part; (3) extracting from the second media component of thewatermarked multimedia object a second extracted watermark part; (4)combining the first extracted watermark part with the second extractedwatermark part; and (5) comparing the combined first and secondextracted watermark parts with the provided watermark.

In a second aspect of the invention, a method is provided for protectinga multimedia object having a first media component and a second mediacomponent, comprising the steps of: (1) providing a watermark byobtaining a signature of the multimedia object, and generating thewatermark as a function of the signature; (2) splitting the watermarkinto a first part and a second part; (3) inserting the first part of thewatermark into the first media component; (4) inserting the second partof the watermark into the second media component; and (5) outputting awatermarked multimedia object.

Where the watermark is provided by obtaining a signature of themultimedia object, the watermarked multimedia object may beauthenticated by: (1) receiving the watermarked multimedia object; (2)extracting from the first media component of the watermarked multimediaobject a first extracted watermark part; (3) extracting from the secondmedia component of the watermarked multimedia object a second extractedwatermark part; (4) generating a combination watermark by combining thefirst extracted watermark part with the second extracted watermark part;(5) generating a signature watermark that is a function of a signatureextracted from the watermarked multimedia object; and (6) comparing thecombination watermark with the signature watermark.

In a further aspect, the invention includes a system for protecting amultimedia object having a first media component and a second mediacomponent, comprising: (1) a mechanism for splitting a watermark into afirst and a second part; and (2) a mechanism for inserting the firstpart into the first media component, and for inserting the second partinto the second media component. The system may further comprise amechanism for obtaining a signature from the multimedia object, and amechanism for generating the watermark as a function of the signature.

In a further aspect, the invention includes a system for verifying orauthenticating a watermarked multimedia object having a first mediacomponent and a second media component, comprising: (1) a mechanism forextracting a first watermark part from the first media component, andfor extracting a second watermark part from the second media component;(2) a mechanism for combining the first extracted watermark part withthe second extracted watermark part; and (3) a mechanism for comparingthe combined first and second watermark parts with a provided watermark.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 describes a first embodiment of a system for watermarking amultimedia object in accordance with the invention.

FIG. 2 describes a first embodiment of a verification mechanism whichcan verify the watermark of a watermarked object.

FIG. 3 describes a second embodiment of a system for watermarking amultimedia object in accordance with the invention.

FIG. 4 describes a second embodiment of a verification mechanism whichcan verify the watermark of a watermarked object.

DETAILED DESCRIPTION OF THE DRAWINGS

The present invention provides a combined watermark system and methodfor multimedia objects. A “multimedia object,” as described herein, maycomprise one or more different types of media components of unlimitedtype, such as audio, video, fixed images, closed caption data,compressed data, email, etc. A multimedia object may exist in any form,including a data file, data signal, data stream, data object, datastructure, transmission, or program. A multimedia object may exist in adigital format or analog format.

Overview

For the purposes of illustration, the following overview of theinvention considers a multimedia object that comprises a video componentand an audio component. However, it is understood that the invention isnot limited to only audio-video applications. If V represents a videoframe, A represents an audio frame, and w is a watermark, thenindependently watermarked content V′ and A′ could be obtained asfollows:V′=V+w, and A′=A+w.As discussed above, such a method provides an opportunity for watermarkidentification and tampering, since for example, subsequent frames of avideo image may be analyzed to reveal the watermark w. In the presentinvention, the robustness of watermark w is greatly enhanced bysplitting it into two parts, w₁ and w₂. The splitting is performed byany function F such that given the two parts w₁ and w₂, the originalwatermark w can be uniquely and exactly computed. The splitting andinserting of the watermarks into the audio and the video frames areaccomplished as follows:w=F(w ₁ , w ₂),  (1)V′=V+w ₁, and A′=A+w ₂.

Since the watermark w is split into two parts and inserted into theaudio and video frames, a potential hacker would have to decipher bothparts of the watermark. Thus, even though successive video frames areoften relatively unchanged, the corresponding audio is continuously andindependently varying, thereby creating a more complex watermark that ismuch more difficult to decipher. In certain circumstances, the potentialhacker would have to determine how the watermark parts w₁ and w₂ shouldbe combined to arrive at the original watermark w.

As noted, the watermark w may be split into parts w₁ and w₂ in anymanner desired (i.e., w=F(w₁, w₂), where F could be any predeterminedfunction.) For example, the watermark may be simply separated into twosmaller parts, i.e., w=w₁+w₂. In such a case, since the amount of datathat can be inserted into video is typically much larger than that whichcan be inserted into audio, the video watermark part may comprise alarger portion of the watermark data. Moreover, in order to make thesystem more robust, parts w₁ and w₂, could be split in a non-contiguousor non-additive manner, thereby making it more difficult for a hacker toidentify the original watermark w. For instance, the system couldutilize logical operations, such as OR, AND, XOR, or variouscombinations of these operations, to split and combine watermarks w₁ andw₂. In order to verify the watermark, the owner could extract the twowatermark parts, and combine them using the predetermined function tocreate the original watermark.

In the case of authentication watermarking, the inserted watermark isdependent on the content to be protected. To achieve this, a signaturethat captures the salient characteristics of the content may beextracted from the multimedia object and then inserted into the contentas a signature watermark. To test the authenticity of the content, thewatermark extracted from the content is compared with the signaturewatermark. If the content is maliciously tampered, then the extractedwatermark and the signature watermark would be different. As in theprevious case, watermarking audio and video independently is not veryrobust and provides opportunities for a potential hacker to extract thewatermark.

To achieve this robustness in applying an authentication watermark,combined audio-video watermarking is performed by first extracting asignature s that is dependent on both the audio and the videocomponents. The function ƒ in the following equation extracts thesalient characteristics of the audio and video. This signature is thensplit into two parts, as in the previous scenario and inserted into boththe audio and video component.s=f(V, A), s=F(w ₁ , w ₂),V′=V+w ₁, and A′=A+w ₂.

In practical cases, the multimedia component might undergo someprocessing after the watermark is inserted. Some examples of theseprocesses include image/video/audio compression, filtering, cropping,etc. It should be recognized that the watermark insertion procedure canbe implemented such that the inserted watermark w can be extracted evenif the content is subjected to these alterations. However, if thealterations are so drastic that they completely alter the visual (orauditory) content of the multimedia component, then it may not bepossible to extract the watermark. Of course, when the multimediacomponent is altered so much that it does not resemble the original, thecontent owner may not be as concerned.

Exemplary Embodiments

Referring now to the figures, FIGS. 1–4 depict various systems forverifying and authenticating multimedia objects. The various devices,mechanisms and systems described therein may be realized in hardware,software, or a combination of hardware and software. They may beimplemented by any type of computer system—or other apparatus adaptedfor carrying out the methods described herein. A typical combination ofhardware and software could be a general purpose computer system with acomputer program that, when loaded and executed, controls the computersystem such that it carries out the methods described herein.Alternatively, a specific use computer, containing specialized hardwarefor carrying out one or more of the functional tasks of the inventioncould be utilized. The present invention can also be embedded in acomputer program product, which comprises all the features enabling theimplementation of the methods and functions described herein, andwhich—when loaded in a computer system—is able to carry out thesemethods and functions. Computer program, software program, program,program product, or software, in the present context mean anyexpression, in any language, code or notation, of a set of instructionsintended to cause a system having an information processing capabilityto perform a particular function either directly or after either or bothof the following: (a) conversion to another language, code or notation;and/or (b) reproduction in a different material form.

1. Watermark Verification

FIG. 1 depicts a system 10 for watermarking a multimedia object 12 tocreate a watermarked multimedia object 22. System 10 comprises a device14 for separating the various media components 16 of multimedia object12. Device 14 may comprise a de-multiplexer or any other hardware orsoftware system for separating the individual media components 16. Mediacomponents 16 may comprise any number of different components. Shown inFIG. 1 are a video component V, audio component A, closed captioncomponent C and miscellaneous signal components S. Components Srepresent any other media component(s) that may be included inmultimedia object 12.

Also contained in system 10 is a predefined watermark 24. A watermarkgenerally comprises some data that can be added to the multimedia objectfor authentication or verification purposes. Watermark 24 may, forexample, comprise the logo of a company or a certificate that has beenassigned to the content owner by a central authority (this is needed toensure that two different content owners do not use the same watermark).Watermark 24 is divided into a plurality of watermark parts 28 by asplitting mechanism 26. Splitting mechanism 26 can split up thewatermark 24 using any predetermined formula as previously described.For example, if the watermark contained n bits of data, a first part ofthe n bits could be inserted into a first watermark part w₁, the nextpart into w₂, the next part into w₃, and the final part into w₄.Watermark parts 28 are then independently added to the media components16 to generate a set of watermarked media components 18. Watermarkedmedia components 18 are then combined back together by system 20 togenerate a watermarked multimedia object 22. Device 20 may comprise amultiplexer or any other hardware or software system for combining themedia components into watermarked multimedia object 22. The resultingwatermarked multimedia object 22 contains independently watermarkedmedia components.

Referring now to FIG. 2, a verification mechanism 11 is shown that canverify the watermark in the watermarked multimedia object 22 created bythe system of FIG. 1. Verification mechanism 11 first divideswatermarked multimedia object 22 into a set of watermarked mediacomponents 19 using device 30. Similar to device 14 shown in FIG. 1,device 30 may comprise a de-multiplexer or any other hardware orsoftware system for breaking the watermarked multimedia object 22 intoindividual watermarked media components 19. The watermarked mediacomponents 19 are then inputted into a watermark extraction mechanism 32that extracts the watermarked parts 34 from each of the watermarkedmedia components 19. Watermarked parts 34 are then combined togetherusing combining mechanism 36 to generate an extracted watermark 38.Combining mechanism 36 combines the watermark parts 34 based on the sameformula that splitting mechanism 26 shown in FIG. 1 used to split theoriginal watermark. The extracted watermark 38 is then compared with theoriginal watermark 24 by comparator 40. The results of the compareoperation are then generated as output 42. If the watermarks matchwithin a certain threshold, then it establishes that the multimediacomponent belongs to the content owner whose watermark appeared in thecontent.

2. Multimedia Object Authentication

Referring now to FIG. 3, a watermark insertion system 45 used forauthenticating purposes is shown, which converts multimedia object 44into a watermarked multimedia object 60. Similar to the embodimentdepicted in FIG. 1, system 45 includes a device 46 for extracting theindividual media components 48 that make up multimedia object 44. Device46 further includes a mechanism for extracting a signature 50 of themultimedia object 44. While device 46 is shown in a single functionalblock, it understood that the system for extracting a signature may befunctionally separated from the system that extracts the mediacomponents 48. Signature 50 contains data that is representative of thedata contained in multimedia object 44. Many different ways ofextracting the signature are known in the art. In this embodiment, thesignature 50 is used as the watermark W. As an alternative, watermark Wcould be a function of the signature 50.

In a manner similar to the embodiment depicted in FIG. 1, the watermarkis then split into a set of watermarked parts 54 by a splittingmechanism 52. Any predetermined formula for splitting the watermark Wcan be used. The individual watermarked parts 54 are then added to themedia components 48 to provide a set of watermarked media components 56.The watermarked media components 56 are then combined using combiningmechanism 58 to generate a watermarked multimedia object 60.

Referring now to FIG. 4, an authentication system 61 is depicted forauthenticating a watermarked multimedia object 60 that includesindependently watermarked media components, such as that created bysystem 45 shown in FIG. 3. System 61 includes a first device 62 forextracting each of the watermarked media components 64. A second device66 is provided for extracting the watermarked parts 68 from each of thewatermarked media components 64. In addition, device 66 extracts thesignature of the received multimedia object, which acts as a signaturewatermark 74. It should be recognized that a separate device could beused to extract the signature of the watermarked multimedia object 60.Signature watermark 74 is extracted using the same formula that was usedto create the original watermark inserted into multimedia object 60, asshown in FIG. 3.

The extracted watermark parts 68 are then combined with combiningmechanism 70 to generate an extracted watermark 72. Combining mechanism70 combines the watermark components 68 using the same ratio, formula orsystem as splitting mechanism 52, shown in FIG. 3, used to split them.The extracted watermark 72 is then compared with the signature watermark74 using comparing mechanism 78. The result of the compare is thengenerated as output 80. If the two watermarks match within a degree ofthreshold, then the multimedia object 60 is authenticated. Conversely ifthe two watermarks 72 and 76 do not match, then it is known that thewatermarked multimedia object 60 is no longer authentic.

The foregoing description of the preferred embodiments of the inventionhave been presented for purposes of illustration and description. Theyare not intended to be exhaustive or to limit the invention to theprecise form disclosed, and obviously many modifications and variationsare possible in light of the above teachings. Such modifications andvariations that are apparent to a person skilled in the art are intendedto be included within the scope of this invention as defined by theaccompanying claims.

1. A method of protecting a multimedia object having a first mediacomponent and a second media component, comprising the steps of:providing a watermark; splitting the watermark into a first part and asecond part; inserting the first part of the watermark into the firstmedia component; inserting the second part of the watermark into thesecond media component; and outputting a watermarked multimedia object.2. The method of claim 1, comprising the further steps of: receiving thewatermarked multimedia object; extracting from the first media componentof the watermarked multimedia object a first extracted watermark part;extracting from the second media component of the watermarked multimediaobject a second extracted watermark part; combining the first extractedwatermark part with the second extracted watermark part; and comparingthe combined first and second extracted watermark parts with theprovided watermark to verify an ownership.
 3. The method of claim 1,wherein the watermark is a signature watermark and is provided by:obtaining a signature of the multimedia object; and generating thesignature watermark as a function of the signature.
 4. The method ofclaim 3, comprising the further steps of: receiving the watermarkedmultimedia object; extracting from the first media component of thewatermarked multimedia object a first extracted watermark part;extracting from the second media component of the watermarked multimediaobject a second extracted watermark part; generating a combinationwatermark by combining the first extracted watermark part with thesecond extracted watermark part; generating a signature watermark thatis a function of a signature extracted from the watermarked multimediaobject; and comparing the combination watermark with the signaturewatermark to authenticate the multimedia object.
 5. A computer programproduct for protecting a multimedia object having a first mediacomponent and a second media component, said product includinginstructions, within a computer-readable medium, executable by aprocessor to carry out acts comprising: splitting a watermark into afirst and a second part; inserting the first part into the first mediacomponents, and inserting the second part into the second mediacomponent.
 6. The product of claim 5, further comprising instructionsexecutable by said processor to carry out acts comprising outputting awatermarked multimedia object, wherein the watermarked multimedia objectincludes the first media component having the first part of thewatermark, and the second media component having the second part of thewatermark.
 7. The product of claim 5, wherein the first media componentis an audio component, and the second media component is a videocomponent.
 8. The product of claim 6, further comprising instructionsexecutable by said processor to carry out acts comprising: obtaining asignature from the multimedia object; and generating the watermark as afunction of the signature.
 9. The product of claim 6, further comprisinginstructions executable by said processor to carry out acts comprising:extracting a first extracted watermark part from the first mediacomponent in the watermarked multimedia object; extracting a secondextracted watermark part from the second media component in thewatermarked multimedia object; combining the first extracted watermarkpart with the second extracted watermark part; and comparing thecombined first and second extracted watermark parts with the watermark.10. The product of claim 8, further comprising instructions executableby said processor to carry out acts comprising: extracting a firstextracted watermark part from the first media component in thewatermarked multimedia object; extracting a second extracted watermarkpart from the second media component in the watermarked multimediaobject; generating an extracted watermark by combining the firstextracted watermark part with the second extracted watermark part;generating a signature watermark that is a function of a signature ofthe watermarked multimedia object; and comparing the extracted watermarkwith the signature watermark.
 11. A computer program product forauthenticating a watermarked multimedia object having a first mediacomponent and a second media component, said product includinginstructions, within a computer-readable medium, executable by aprocessor to carry out acts comprising: extracting a first watermarkpart from the first media component; extracting a second watermark partfrom the second media component; combining the first extracted watermarkpart with the second extracted watermark part; and comparing thecombined first and second watermark parts with a provided watermark. 12.The product of claim 11, wherein the provided watermark is generated asa function of a signature of the watermarked multimedia object.
 13. Theproduct of claim 11, wherein the first media component is a videocomponent and the second media component is an audio component.
 14. Theproduct of claim 13, wherein the watermarked multimedia object has athird media object, and wherein the third media object is a closedcaption component.
 15. A computer system comprising the product of claim11.
 16. A computer system comprising the product of claim 5.